Nerd time with Nibbles

"Nibbles" (nibbles) 07/26/2018 at 23:24 • Filed to: None

6

27

My primary function at work is making machines do the work of people. Over the past few years I have increased efficiency and lowered opportunity for human error by orders of magnitude by writing programs to do the things I and my coworkers would otherwise spend tedious hours clicking and typing and watching.

One particular process had eluded me though. Some times records have to be re-sent. Often times those record counts are in the millions and the gatekeeper system can only handle ~10k records per hour. This would lead to me staying up for days on end, watching a queue count email received every 30 minutes, and manually triggering resends based on the thresholds

Not any more! On Tuesday one of my service accounts was granted SELECT access to the Oracle database where the queue count resides. Yesterday I built a .NET application to query that database for relevant data and automatically trigger record pushes based on thresholds determined by the business. Today I used it to perform its duties.

And perform it has! It’s been resending records for six hours without a hiccup. This is a small batch, so only 12 hours of processing is expected. We even had an hour where queue count went above the threshold - due to regular data movement adding to the load, not because the system failed to perform as expected - and the system waited patiently until the queue was happy and continued on its merry way.

The best part? No actual testing was performed before execution today. I had no clue if it was going to function and here it is, functioning flawlessly. I like what I do. 

Just Jeepin' > Nibbles 07/26/2018 at 23:47

3

I once automated what I thought was the least interesting work I could imagine, but my team didn’t like it. I think they preferred the slow, mindless process vs the relatively simple DSL I created to replace it.

Can’t win them all.

BaconSandwich is tasty. > Nibbles 07/27/2018 at 00:04

1

.Net, eh? I thought you were going to say Python. 

Highlander-Datsuns are Forever > Nibbles 07/27/2018 at 00:05

2

My boss bought a huge bag of m&m’s, that’s a win for me.

wafflesnfalafel > Nibbles 07/27/2018 at 00:08

1

M y current employer’s version of automation - “L ets ship it 7700 miles west of here and let a bunch of 3rd party humans in Mumbai do it manually . Oh, but there is no way to fix all the mistakes they make, so good luck! ”

ttyymmnn > Nibbles 07/27/2018 at 00:09

3

On Tuesday one of my service accounts was granted SELECT access to the Oracle database where the queue count resides. Yesterday I built a .NET application to query that database for relevant data and automatically trigger record pushes based on thresholds determined by the business.

Seriously, though, congratulations . This sort of techie stuff is so far over my head.

AM3R, lost another burner > Nibbles 07/27/2018 at 00:35

1

Great job man!

AfromanGTO > Nibbles 07/27/2018 at 01:12

1

Good job man! Keep up the good work!

Nibbles > BaconSandwich is tasty. 07/27/2018 at 02:18

0

Hah I don’t know one lick of Python

Nibbles > wafflesnfalafel 07/27/2018 at 02:18

0

My boss wants me to offshore some of my work too. My hesitance is well known

random001 > ttyymmnn 07/27/2018 at 07:12

0

This....

nerd_racing > Nibbles 07/27/2018 at 08:07

3

I do a lot of torsional simulations, one of our customers wanted specific data that I couldn’t get in my usually frequency based differential equations simulations.  So I built a time domain model to simulate a 30 second linear acceleration dyno sweep.  It took a couple days, but it works.  I can even add sensors to output data for other things that were super tedious.  I totally appreciate your breakthrough! 

functionoverfashion > Nibbles 07/27/2018 at 09:51

2

That is so satisfying. I will spend many hours and/ or days creating and refining tools to save minutes, seconds, or hours on repetitive tasks. When I see a coworker manually counting things in a spreadsheet, which was exported from a system we use that has a very sophisticated reporting tool using a semi-custom SQL language, “T here has to be a better way...” and off I go.

I mean seriously someone was looking at a computer screen with an open spreadsheet going “one, two, three...” 

Nibbles > functionoverfashion 07/27/2018 at 10:54

2

Wow. Just...

They don’t know Excel has a count function?

But yes all of this. Some people say I’m automating people out of jobs. Hell, even I say that. But the truth is I’m making everyones’ lives easier, lessening risk, and keeping patient treatment impact at a complete minimum. I love what I do. I spent 15 years treading water  in desktop support until this gig came around. I’ll never look back

Nibbles > nerd_racing 07/27/2018 at 10:55

0

That is awesome!

functionoverfashion > Nibbles 07/27/2018 at 11:01

2

Interesting; I do desktop support half-time at a university, and I like the variety and flexibility, as well as walking around all over campus and talking to people; also, you know, troubleshooting is interesting.

I’m definitely not hoping to do this for 15 years though. It’s nice while I have young kids because there are 2-3 other guys who also do it and I can work basically 9-3 so I can be with my kids outside of school hours.

And I mean, yeah. Excel has like, at least 10 ways to do what they were doing. But backing up, the file came from a system that has reporting built in. Like, no SQL knowledge required - which is good because my knowledge is rudimentary at best. I just know the data, and can figure the rest out as I go.

Welp. Cheers.

Nibbles > functionoverfashion 07/27/2018 at 12:18

4

I enjoyed my time in desktop support for the first few years. I got my start at Ball Aerospace and got to get my hands on really neat things! I was on the Kepler team, I saw the Mars Rovers (Opportunity, Curiosity ) being built and tested, even some DoD programs I still can’t talk about :). But as the years ran by I did get worn out. I went into hardware/firmware engineering for HP Enterprise (then Hewlett/Packard), got some sysadmin experience, then tried supervising. Finally I landed at my current company as a field server admin; I saw how painful and tedious 95% of the work was, so I started automating it. The business saw my efforts and rewarded me graciously; I’m now the Senior Software Configuration Management Engineer, looking at an architect role in the near future.

TheRealBicycleBuck > Nibbles 07/27/2018 at 12:38

2

I LOVE it when automation takes over and works as intended! I have to do some in my job, but rarely does it involve direct selects from a database. I usually have to batch process geospatial tasks (select from a set based on the geospatial relationship to another set, then process data and output to another set for yet another operation).

The latest was an analysis of bridges. During high-water events like Hurricane Harvey, debris can pile up on the bridge and force the current deeper than normal, causing erosion in the drainage channel which may get deep enough to uncover the pilings (columns) which hold up the bridge. With over 800 bridges in our area, I had to set up tools to analyze bridge, stream flow, and debris removal data to determine which bridges are at greatest risk. I automated the task with a model, then batched the model to iterate over all 800 bridges. Now that we know which ones are at risk, we can prioritize which one should be inspected by our bridge engineers first.

We have some nifty tools which allow us to set up the model visually, test them to confirm they work as expected, then convert everything over to Python so we can recode them for batch processing and to work as standalone tools.

Fun stuff.

TheRealBicycleBuck > ttyymmnn 07/27/2018 at 12:54

2

I shall translate!

Service account - an account which is used to connect to and interact with a database. Much like your user account allows you to interact with Kinja.

SELECT - a database command which allows the service account to choose records in a database which match specified criteria. For example, I might SELECT from the Kinja database any USERS which have more than 100 posts.

Queue - think of this as a waiting area in a database. It’s used for a lot of things, but in this case, it’s where a line of commands are waiting for their turn to be processed.

.NET - in simple terms, a programming language. In this case, an application (program) written in the .NET language.

Query - ask a question. In this case, it’s a question written in .NET which SELECTs a set of records in the database.

What he’s describing is a program that periodically queries the database to see how many records have been added or how many have changed status . When the count gets high enough, it performs some action. It’s like painting car parts. You don’t want to go through the effort of mixing the paint until you have enough parts to justify the work. The businesses that use his program can adjust the threshold based on their business practices. Some might process every 100 records, some might wait until there are 10,000 records.

TheRealBicycleBuck > Nibbles 07/27/2018 at 12:58

0

I don’t know diddly about .NET or other object-based languages. I’m old school - DOS, VB, and now, Python.

functionoverfashion > Nibbles 07/27/2018 at 13:16

1

That’s great, it sounds like you’ve found a perfect path and are all over it! Congrats.

My path is less clear, but I am confident. This job was a total left turn from my previous General Manager position of a small business, so... who knows?

Nibbles > functionoverfashion 07/27/2018 at 13:54

1

You’re young(er, I take it), you have  time. Don’t get stuck in a career you hate

Nibbles > TheRealBicycleBuck 07/27/2018 at 13:55

0

If you know VB then C# is a logical transition. And it’s fun!

functionoverfashion > Nibbles 07/27/2018 at 13:58

0

I’m 37. Still plenty of time, the way I see it. And I mostly like what I do. My kids take up a lot of my time, and I’m good with that.

TheRealBicycleBuck > Nibbles 07/27/2018 at 14:12

2

I’ve thought about adding it to my quiver for some time. But it’s one of those need vs want situations. Right now my focus is on the “big data” tools in Python. I’m learning numpy, pandas, matplotlib and seaborn plus the tools to connect to and query SQL from python. Fun stuff, for sure.

Oh, I’m also working on pilot ground school. That’s taking a ton of my spare time. Learning is fun, fun, fun!  

ttyymmnn > TheRealBicycleBuck 07/27/2018 at 15:26

1

Thanks. I gleaned a certain sense of all that, so my comment wasn’t entirely in earnest. My brain just doesn’t work that way. At all.

wafflesnfalafel > Nibbles 07/27/2018 at 21:42

0

I t is super tough - I am really uncomfortable giving up key processes to a 3rd party you have virtually no control over. We are on our 3rd attempt in the last 10 years - the first two tries were disastrous and expensive.

BaconSandwich is tasty. > Nibbles 07/27/2018 at 22:51

1

I had forgotten you were a .Net person. I started with .Net and still love it. Been doing Python for the last four years or so. I do miss .Net on occasion.